Skip to content

Increase category/tag pool and fix search-and-choose output quality#4417

Merged
lucaslyl merged 8 commits intomainfrom
log-on-linkcateg-and-linktag-in-realm-staging-why-always-have-problem
Apr 17, 2026
Merged

Increase category/tag pool and fix search-and-choose output quality#4417
lucaslyl merged 8 commits intomainfrom
log-on-linkcateg-and-linktag-in-realm-staging-why-always-have-problem

Conversation

@lucaslyl
Copy link
Copy Markdown
Contributor

linear: https://linear.app/cardstack/issue/CS-10801/increase-categorytag-pool-and-fix-search-and-choose-output-quality

The Problem (Before):

❌ Search-and-choose had a critical hallucination issue:

  • You pass the LLM a list of categories/tags to choose from
  • LLM returns results that are NOT in that list
  • LLM hallucinates IDs/names from its training data instead of picking from your actual options
  • Output is confusing and unreliable, that will the AI autolinkCateg/autolinkTag is actually work, but the result is no relevant for that card, that why the categ/tag is not linked.

Example: You give it tags: ["App", "Dashboard", "Form"] → LLM returns "XYZ-unknown-tag" (not in list)

Fix:

  1. Numbered Index Pattern (in search-and-choose)
  • Switched to numbered index pattern instead of raw IDs
  • Prevents AI from making up results outside the provided list
  • Instead of: ["../Tag/app", "../Tag/dashboard", "../Tag/form"] (raw names/IDs)
  • Use: [1. App, 2. Dashboard, 3. Form] (numbered list)
  • Result: LLM just returns numbers [1, 2] instead of hallucinating new names
  • ✅ More accurate, fewer hallucinations
  1. Expand Tag Pool (add Skill tag, remove card-type)
  • Give AI more relevant choices to pick from ,else it will return []
  • Instead of AI returning [] (nothing found), it has better options
  • Better categorization = better selection

3.Add Fallback Strategy for tags section

  • If AI can't find relevant tag → try Source/Origin tags instead
  • Instead of: "No tag found → return []"
  • Now: "No specific tag → use Official/User Contributed/Bundled as fallback"
  • ✅ Prevents empty results, always finds something appropriate
Screen.Recording.2026-04-16.at.2.22.55.PM.mov

);
console.warn(
`[SearchAndChoose:${candidateTypeCodeRef.name}] parsed indices:`,
selectedIndices,
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left console.warn here for better debugging in both localhost and production for future

@lucaslyl lucaslyl changed the title Log on linkcateg and linktag in realm staging why always have problem Increase category/tag pool and fix search-and-choose output quality Apr 16, 2026
@lucaslyl lucaslyl requested review from a team April 16, 2026 06:28
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 688f493c24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/catalog-realm/crm-app/Tag/8a14f671-4daf-4195-aca5-85dc143d15b8.json Outdated
@lucaslyl
Copy link
Copy Markdown
Contributor Author

@codex review

@github-actions
Copy link
Copy Markdown

Preview deployments

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7f469a6baf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +157 to +160
return parsed.filter(
(v): v is number =>
typeof v === 'number' && Number.isInteger(v) && v > 0,
);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Accept quoted indices from LLM selection output

parseIndicesFromLlmOutput drops any value that is not a numeric JSON literal, so common responses like ["1"] are treated as no selection even though they unambiguously point to a valid option. In this commit, listing-create depends on search-and-choose for tag/category linking, so these quoted-number responses now degrade to empty auto-link results and trigger fallback behavior instead of selecting the intended candidate.

Useful? React with 👍 / 👎.

Comment on lines 183 to 186
oneShotLogger.debug('raw llm response meta', {
status: result.response.status,
model: input.llmModel || 'anthropic/claude-3-haiku',
model: input.llmModel || 'anthropic/claude-haiku-4.5',
usage: responseData.usage || null,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 Badge Log the same default model that is actually requested

The request payload still defaults to anthropic/claude-3-haiku, but this new log line reports anthropic/claude-haiku-4.5 when input.llmModel is unset. That makes production debugging and usage/cost attribution unreliable because telemetry says a different model than the one sent to OpenRouter.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid here, either we shall update the payload as well

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 16, 2026

Host Test Results

2 265 tests  +19   2 250 ✅ +19   2h 31m 47s ⏱️ -49s
    1 suites ± 0      15 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit 1e0eb02. ± Comparison against base commit b16dc99.

This pull request removes 1 and adds 20 tests. Note that renamed tests count towards both.
Chrome ‑ Integration | commands | search-and-choose: uses sourceContextCodeRef source instead of the searched type source
Chrome ‑ Integration | Command | host command schema generation test > command schema generation: getInputJsonSchema for EvaluateModuleCommand
Chrome ‑ Integration | Command | host command schema generation test > command schema generation: getInputJsonSchema for InstantiateCardCommand
Chrome ‑ Integration | commands | evaluate-module: module with broken import fails evaluation
Chrome ‑ Integration | commands | evaluate-module: valid module passes evaluation
Chrome ‑ Integration | commands | instantiate-card: containsMany field with non-array value fails instantiation
Chrome ‑ Integration | commands | instantiate-card: valid card with instance data passes instantiation
Chrome ‑ Integration | commands | instantiate-card: valid card with no instance data passes instantiation
Chrome ‑ Integration | commands | search-and-choose: LLM index selection uses sourceContextCodeRef as context
Chrome ‑ Integration | commands | search-and-choose: parseIndicesFromLlmOutput accepts numeric literals
Chrome ‑ Integration | commands | search-and-choose: parseIndicesFromLlmOutput accepts quoted numeric strings
…

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 16, 2026

Realm Server Test Results

  1 files  ±0    1 suites  ±0   14m 14s ⏱️ + 1m 8s
894 tests ±0  894 ✅ ±0  0 💤 ±0  0 ❌ ±0 
966 runs  ±0  966 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 1e0eb02. ± Comparison against base commit b16dc99.

♻️ This comment has been updated with latest results.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the reliability of LLM-driven “search-and-choose” by switching from free-form ID selection to numbered index selection, and expands/standardizes the catalog category/tag corpus so auto-linking has better options.

Changes:

  • Updated search-and-choose prompting + parsing to use numbered options and numeric index outputs (with new parsing tests).
  • Improved listing-create background autopatch resilience by using Promise.allSettled and logging per-task failures.
  • Expanded/normalized catalog realm metadata (new tags/categories/spheres, plus updated listing relationships) and added a catalog structure reference doc.

Reviewed changes

Copilot reviewed 129 out of 134 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
packages/host/tests/integration/commands/search-and-choose-test.gts Update integration expectations for numeric index output; add parser unit tests
packages/host/app/commands/search-and-choose.ts Switch prompt to numbered options + numeric parsing; update selection mapping and logging
packages/host/app/commands/one-shot-llm-request.ts Update default model string used for OpenRouter requests/logging
packages/host/app/commands/listing-create.ts Use Promise.allSettled for background autopatch and refine tag/category selection prompts
packages/catalog-realm/sprint-planner/Tag/e3625690-2231-428e-8f6d-d123d8f5cc6c.json Update tag metadata structure (cardInfo/relationships normalization)
packages/catalog-realm/sprint-planner/Tag/631d1b5d-fcd0-465c-964e-e535fc6bb893.json Update tag metadata structure (cardInfo/relationships normalization)
packages/catalog-realm/crm-app/Tag/8a14f671-4daf-4195-aca5-85dc143d15b8.json Update tag metadata structure (cardInfo/relationships normalization)
packages/catalog-realm/catalog-app/catalog_structure.md Add documentation describing spheres/categories/tags and their relationships
packages/catalog-realm/ThemeListing/f87667d9-7616-4baf-b2fe-8ce5d393f676.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/ThemeListing/4d6390f1-3bae-4e75-a95d-4309010e03e4.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/Tag/user-contributed.json Normalize tag color + cardInfo.name
packages/catalog-realm/Tag/tracker.json Add new “Tracker” content-type tag
packages/catalog-realm/Tag/theme.json Normalize tag color/meta/cardInfo.name
packages/catalog-realm/Tag/template.json Add new “Template” content-type tag
packages/catalog-realm/Tag/skill.json Add new “Skill” content-type tag
packages/catalog-realm/Tag/report.json Add new “Report” content-type tag
packages/catalog-realm/Tag/poster.json Normalize tag color/meta/cardInfo.name
packages/catalog-realm/Tag/planner.json Add new “Planner” content-type tag
packages/catalog-realm/Tag/official.json Normalize tag color + cardInfo.name
packages/catalog-realm/Tag/new.json Add new “New” source/origin tag
packages/catalog-realm/Tag/general.json Add new “General” source/origin fallback tag
packages/catalog-realm/Tag/game.json Normalize tag color + cardInfo.name
packages/catalog-realm/Tag/form.json Add new “Form” content-type tag
packages/catalog-realm/Tag/featured.json Add new “Featured” source/origin tag
packages/catalog-realm/Tag/dashboard.json Add new “Dashboard” content-type tag
packages/catalog-realm/Tag/community.json Add new “Community” source/origin tag
packages/catalog-realm/Tag/card.json Add new “Card” content-type tag
packages/catalog-realm/Tag/calculator.json Normalize tag color + cardInfo.name
packages/catalog-realm/Tag/bundled.json Normalize tag color + cardInfo.name
packages/catalog-realm/Tag/app.json Add new “App” content-type tag
packages/catalog-realm/Tag/ai.json Normalize tag naming (“AI”) and set source/origin color
packages/catalog-realm/Sphere/work.json Add WORK sphere instance
packages/catalog-realm/Sphere/play.json Add PLAY sphere instance
packages/catalog-realm/Sphere/life.json Add LIFE sphere instance
packages/catalog-realm/Sphere/learn.json Add LEARN sphere instance
packages/catalog-realm/Sphere/build.json Add BUILD sphere instance
packages/catalog-realm/SkillListing/talk-like-a-pirate.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/SkillListing/cursor-coding.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/FieldListing/geo-search-point.json Switch category relationship from UUID to canonical category slug
packages/catalog-realm/FieldListing/geo-point.json Switch category relationship from UUID to canonical category slug
packages/catalog-realm/FieldListing/fb9494c4-0d61-4d2d-a6c0-7b16ca40b42b.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/Category/web-development.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/video-games-interactive.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/ui-components-design.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/travel-lifestyle.json Fill missing cardInfo fields and switch sphere relationship to canonical sphere slug
packages/catalog-realm/Category/travel-experiences.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/technical-documentation.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/teaching-instruction.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/sports-fitness.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/software-development.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/social-networking.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/skills-training.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/shopping-consumer.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/security-privacy.json Add new category and assign it to BUILD sphere
packages/catalog-realm/Category/science-discovery.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/research-knowledge.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/real-estate-property.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/reading-literature.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/quality-assurance.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/prototyping-experiments.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/project-management.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/professional-services.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/product-catalogs.json Fill missing cardInfo fields and switch sphere relationship to canonical sphere slug
packages/catalog-realm/Category/personal-finance.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/parties-celebrations.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/operations-supply-chain.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/news-media.json Add new category and assign it to LIFE sphere
packages/catalog-realm/Category/marketing-growth.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/maps-navigation.json Add new category and assign it to LIFE sphere
packages/catalog-realm/Category/legal-compliance.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/language-learning.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/knowledge-management.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/hr-people-management.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/home-living.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/hobbies-crafts.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/health-wellness.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/hardware-iot.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/goals-habits.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/gaming.json Add new category and assign it to PLAY sphere
packages/catalog-realm/Category/food-cooking.json Add new category and assign it to LIFE sphere
packages/catalog-realm/Category/family-relationships.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/events-celebrations.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/entertainment-media.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/education-courses.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/e-commerce-online-sales.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/devops-infrastructure.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/developer-tools-code.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/design-creative.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/data-management.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/data-engineering.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/data-analytics.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/customer-support.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/creative-projects.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/content-creation.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/communication.json Add new category and assign it to WORK sphere
packages/catalog-realm/Category/certifications-training.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/career-development.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/business-development.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/api-integration.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/analytics-reporting.json Switch sphere relationship from UUID to canonical sphere slug
packages/catalog-realm/Category/ai-automation.json Fill missing cardInfo.name and switch sphere relationship to canonical sphere slug
packages/catalog-realm/Category/adventure-outdoor-activities.json Fill missing cardInfo.name and switch sphere relationship to canonical sphere slug
packages/catalog-realm/Category/accounting-finance.json Fill missing cardInfo.name and switch sphere relationship to canonical sphere slug
packages/catalog-realm/CardListing/f0c0ad91-0194-46b9-a971-9e60d637a51a.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/f0277f2a-26af-4e25-8222-6d6492bc08f6.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/ef22ef9b-d7f6-4593-8bec-e2cc283103ec.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/e7fe0e9a-f6ce-40f7-bf63-02157530cae1.json Switch tag/category relationships from UUIDs to canonical slugs
packages/catalog-realm/CardListing/daily-report-dashboard.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/d5208656-b610-4ad5-81f8-cbd7d512493a.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/cbe2c79b-60aa-4dca-bc13-82b610e31653.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/b920054b-0f48-440b-b50e-64a5a097e7a5.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/a51b73a3-08c5-4dfd-b22a-8e66a0285801.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/a487a65f-4356-42a1-9da0-c6e0be0166e4.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/9ffcb030-9c5a-4d17-ace6-2ea83c6ae2d4.json Switch tag/category relationships from UUIDs to canonical slugs
packages/catalog-realm/CardListing/9de5efec-83f7-42d3-9425-3a19e8a15fd2.json Switch category relationship from UUID to canonical category slug
packages/catalog-realm/CardListing/9b84fcb0-80f4-456e-9fb3-e9abd0ec3fcb.json Switch category relationship from UUID to canonical category slug
packages/catalog-realm/CardListing/8190ab54-5b6f-457b-8c87-43f09eabf757.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/79937c95-92ec-4393-acbb-b1028b78defe.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/694bf025-0904-488d-af1b-2b88bbd8df43.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/61be974d-6254-4ddd-9917-d91fa165a536.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/5b6c4561-ca61-4e38-aeb1-7d9f7ff9778e.json Switch category relationship from UUID to canonical category slug
packages/catalog-realm/CardListing/4aca5509-09d5-4aec-aeba-1cd26628cca9.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/49ea1457-ecc8-418d-9013-d74730b116e1.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/36de56b9-9ee7-4105-8f2c-497819d76da6.json Switch tag/category relationships from UUIDs to canonical slugs
packages/catalog-realm/CardListing/358a08b8-9e03-4a39-a40b-539eb3e01622.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/2d838726-ed2c-437d-a171-eade2544ca66.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/CardListing/2b1f1edf-dc57-4198-a180-9b07fc26dbfe.json Switch category relationship from UUID to canonical category slug
packages/catalog-realm/CardListing/2669572f-00e1-485f-900d-5b686acfe9c6.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/CardListing/10ff8743-4b8c-403e-95a6-90c36cbb98ca.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/AppListing/c1413482-092a-45b4-b92b-d8d2672641e7.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/AppListing/a47dcb12-f1aa-4415-8be1-5ae04f5db457.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/AppListing/95cbe2c7-9b60-4afd-8a3c-1382b610e316.json Switch tag relationships from UUIDs to canonical tag slugs
packages/catalog-realm/AppListing/77083eb2-d809-4366-ad85-f757dc5c693b.json Switch tag relationship from UUID to canonical tag slug
packages/catalog-realm/AppListing/707a0c91-4c88-45d9-a28e-1195c8834b66.json Switch tag relationship from UUID to canonical tag slug

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +52 to +58
```
adoptsFrom: {
"module": "../catalog-app/listing/sphere", // For Sphere cards
"module": "../catalog-app/listing/category", // For Category cards
"module": "../catalog-app/listing/tag", // For Tag cards
"module": "../catalog-app/listing/listing" // For Listings (CardListing, AppListing, etc.)
}
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The adoptsFrom example shows an object with four repeated "module" keys, which isn’t valid JSON and can mislead readers. Consider rewriting this section as four separate examples (Sphere/Category/Tag/Listing) or as a list of { module, name } pairs rather than a single JSON object.

Copilot uses AI. Check for mistakes.
Comment on lines +178 to 185
private formatCandidatesAsNumberedList(instances: any[]): string {
return instances
.filter((c) => c && c.id)
.map((c) => {
const title = c.title || '';
.map((c, i) => {
const name = c.cardInfo?.name || c.name || '';
const summary = c.cardInfo?.summary || '';
return summary
? `${c.id} :: ${title} — ${summary}`.trim()
: `${c.id} :: ${title}`.trim();
return summary ? `${i + 1}. ${name} — ${summary}` : `${i + 1}. ${name}`;
})
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

formatCandidatesAsNumberedList() uses c.cardInfo?.name || c.name for the display label, but search results return instances with cardTitle (see SearchCardsByQueryCommand building summaries from c.cardTitle). This will often produce blank option labels in the prompt, reducing selection quality. Consider preferring c.cardTitle (and/or falling back to c.id) for the numbered list label.

Copilot uses AI. Check for mistakes.
Comment on lines +103 to +111
const validInstances = instances.filter((c: any) => c && c.id);
const selectedIndices = this.parseIndicesFromLlmOutput(
res.output || '[]',
).slice(0, max);
const selectedCards = selectedIndices
.map((i) => validInstances[i - 1])
.filter(Boolean);
const selectedIds = selectedCards.map((c: any) => c.id);

Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

selectedIndices is taken directly from the LLM output and can include duplicates; the current mapping will then duplicate selectedCards/selectedIds, even though the prompt asks for no duplicates. Consider normalizing by de-duping indices (while preserving order) before mapping to instances.

Copilot uses AI. Check for mistakes.
Comment on lines +112 to +124
// Log a warning if the LLM output could not be parsed into valid selections, to aid debugging
if (selectedCards.length === 0) {
console.warn(
`[SearchAndChoose:${candidateTypeCodeRef.name}] result is empty. candidates sent to LLM:\n${numberedCandidates}`,
);
console.warn(
`[SearchAndChoose:${candidateTypeCodeRef.name}] LLM raw output: "${res.output}"`,
);
console.warn(
`[SearchAndChoose:${candidateTypeCodeRef.name}] parsed indices:`,
selectedIndices,
);
}
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warns whenever selectedCards.length === 0, but an empty selection is an expected outcome (the prompt explicitly allows returning []). This can create noisy logs and can dump potentially large candidate lists / LLM output into logs. Consider only warning when the LLM output is non-empty/non-[] but fails to parse, or when indices are out of range, and avoid logging full candidate text unless in a debug logger.

Copilot uses AI. Check for mistakes.
Comment on lines +96 to 101
const res = await oneShot.execute({
systemPrompt,
userPrompt,
llmModel: llmModel || 'anthropic/claude-3-haiku',
llmModel: llmModel || 'anthropic/claude-haiku-4.5',
codeRef: selectionContextCodeRef ?? candidateTypeCodeRef,
});
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SearchAndChooseCommand hard-codes a default llmModel value here even though OneShotLlmRequestCommand also has its own default. This duplicates configuration in two places and makes it easier for defaults to drift. Consider passing llmModel through when provided and otherwise leaving it undefined so the one-shot command (or a shared constant) is the single source of truth.

Copilot uses AI. Check for mistakes.
Comment on lines 166 to 173
const result = await sendRequestViaProxyCommand.execute({
url: 'https://openrouter.ai/api/v1/chat/completions',
method: 'POST',
requestBody: JSON.stringify({
model: input.llmModel || 'anthropic/claude-3-haiku',
model: input.llmModel || 'anthropic/claude-haiku-4.5',
messages: generationMessages,
stream: false,
}),
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default OpenRouter model is hard-coded here (and in other commands) as a string. Elsewhere in the host codebase model defaults come from runtime-common constants / Matrix model configuration (e.g. DEFAULT_LLM, DEFAULT_CODING_LLM). Consider sourcing the default from the shared constants/config instead of embedding a model id here, so changing defaults doesn’t require touching multiple call sites.

Copilot uses AI. Check for mistakes.
Comment on lines +156 to +179
@@ -167,8 +167,24 @@ export default class ListingCreateCommand extends HostBaseCommand<
codeRef.module,
codeRef,
),
]).catch((error) => {
console.warn('Background autopatch failed:', error);
]).then((results) => {
const names = [
'autoPatchName',
'autoPatchSummary',
'autoLinkTag',
'autoLinkCategory',
'autoLinkLicense',
'autoLinkExample',
'linkSpecs',
];
Copy link

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The names array must stay in the exact same order as the Promise.allSettled([...]) list above; if either list changes later, the log labels can silently become incorrect. Consider pairing each promise with its name (e.g., an array of { name, promise }) so the association is structural rather than positional.

Copilot uses AI. Check for mistakes.
@lucaslyl lucaslyl merged commit 44d285c into main Apr 17, 2026
66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants